The immediate impact of the cargo decline affects virtually every business around the ports, but port officials say this downturn will soon be felt much more broadly.