Jimmy played a single game for the Norwich City reserves on 15th May 1997 in a 4-1 defeat at home to Luton. At the time he was a regular in Chelsea's youth team. James then joined Fulham where he was one of Mickey Adams' last signings before the arrival of Kevin Keegan and Ray Wilkins. He obviously did not fit in with the ambitious plans of Keegan and Al-Fayed and was released in the summer of 1998.

He had a trial with Shrewsbury before joining Torquay United on a one month's contract on October 29th 1998 to cover for an injury crisis with their central defenders. He made his debut for Torquay in their 1-1 draw with Mick Wadsworth's Scarborough on Halloween. His impressive form for Torquay earned him a contract until the end of the 1998/99 season. According to the South London Press of 24th March 1999, Jimmy also interested Charlton manager Alan Curbishley who was said to be preparing a £350,000 bid this summer for him.

At the end of August 2001, Jimmy was transfer-listed by Torquay despite making almost 100 league appearances for them. It was rumoured that local rivals Exeter are keen to snap him up. In September 2001 fellow South-West side Yeovil expressed an interest in taking Jimmy on loan with view to a permanent transfer if it worked well. Jimmy then had an unsuccessful trial with Rushden and Diamonds at the beginning of October 2001. Jimmy left Plainmoor permanently on December 10th 2001 and joined Dover a few weeks later.

Jimmy returned to the West Country on September 6th 2002 signing for Dr Martens Western Division side Taunton Town. By the beginning of October 2002, Jimmy had moved again, joining Harrow Borough of the Ryman League. At the beginning of December 2002, the Non League Paper reported that Jimmy had signed for Yeovil.

Yeovil won the Conference Championship on April 12th 2003 when rivals Chester could only draw with Woking. Jimmy had made just 10 appearances prior to the championship winning weekend and so the club decided he was not of the standard required for Division Three and released him at the beginning of May 2003.

In July 2003, Jimmy played on trial for Welsh side TNS in their 10-0 thrashing of Malpas in preparation for their UEFA Cup clash with Manchester City. After some "commanding performances in central defence" in these pre-season friendlies, Jimmy completed his move to the Welsh side.

After nearly two seasons at Treflan, Jimmy left Vauxhall Masterfit Retailers Welsh Premier title-contenders Total Network Solutions The club agreed to cancel Aggrey's contract by mutual consent with the central defender later signing for Irish club Bohemians. In June 2005, Jimmy signed an eighteen month contract with Woking.

In early February 2006, Jimmy was loaned to Southport.

He subsequently left and retired from professional football to pursue an acting career. He played Willam Laurent Dioup in Sky One's Dream Team.