In general, it is hard to determine the network structure because it is related to the generalization ability. Moreover, it is also hard to analyze networks trained by back propagation learning. In order to solve these problems, structural learning with forgetting (SLF) has been proposed. In this paper, we improve SLF in terms of structuring ability, and propose parallel multi-layer networks. Using our method, (1) wastefully distributed representation of hidden units are suppressed without revival of unnecessary parameters, (2) forgetting is accelerated, (3) network structure is automatically determined, and (4) classification rules are extracted in a discrete valued inputs problem and a continuous valued one. This method is applied to the XOR problem and the thyroid function classification as a practical problem of continuous valued inputs. It is found that our method is twice faster than SLF and its success rate is about 100% in terms of obtaining the smallest number of hidden units.