Labor Secretary Tom Perez on Tuesday blamed Congress for the delays his department has faced in approving visas for foreign agricultural workers, and rejected the idea that Labor is simply dragging its feet.

Rep. Andy Harris, R-Md., said the workers were crucial to his state’s seafood industry, and cited reports from last month that said it was taking steadily longer for the Labor Department to approve the special H-2B visas for those workers. Perez confirmed this, conceding, “Frankly, we fell significantly behind… It absolutely got worse.”