Contractor withdrawn from housing project for sugar workers