The 88-year-old patriarch is suffering from an undisclosed terminal illness and his heath has taken a turn for the worse in the last week, according to his son Jermaine.
He has been ill for a number of months and has banned the family from seeing him, he added.
It is understood that the music manager's wife Katherine and daughters Rebbie and Joh'Vonnie along with granddaughter Yashi Brown visited Joe in Las Vegas, Nevada on Tuesday.
Speaking to the Mail, Joe's son Jermaine said: “No one knew what was going on – we shouldn't have to beg, plead, and argue to see our own father, especially at a time like this.
“We have been hurting. “We were not being told where he was and couldn’t get the full picture. Even from the doctor. My mother was worried sick.”
Jermaine, 63, added: “He’s very very frail, he doesn't have long. The family needs to be by his bedside – that’s our only intention in his final days. “It’s what any family would want, but some people around him think they know better...and they don’t.”
It is believed Joe himself gave verbal instructions – which has been recorded on a phone – making clear he didn't want visitors or anyone from the family to see him or his medical records.
A source close to the camp said the people around Joe are those who have been there for him in his everyday life. They said: "They are acting in his interests and no one else's.
"The Jackson family doesn't have the capacity to deal with this on its own, so certain people are taking charge of the situation.
"It's delicate... and not as black and white as people are making out." It was Joe who guided his children to unprecedented showbiz success as The Jackson 5.