Recent studies have shown that deep convolutional neural networks (CNNs) have been successfully used for texture representation and recognition. One of the most successful texture recognition methods is the deep texture encoding network (DeepTEN), which has been shown to be effective. However, this network directly uses redundant CNN features with generality and ignores the role of multiorder information during the encoding and learning processes. To address these issues, this paper proposes a double-order knowledge fusion and attentional encoding network for texture recognition (DFAEN). First, crucial texture features are encoded by an embedded attention mechanism. Second, double-order modeling is implemented in the encoding and learning stage to make full use of convolution feature information with different orders, enabling the network to focus on and learn more texture domain information. Our method can stably and effectively perform end-to-end optimization. Evaluation experiments conducted on several widely used benchmark datasets (e.g., the FMD, MINC-2500, the DTD, KTH-TISP-2b, and GTOS-mobile) show that our method clearly demonstrates superior performance to that of competing approaches.