Abstract: Relative pose estimation of camera is to calculate the relative position and pose of the camera during imaging. It is a key problem in the field of computer vision, such as image mosaic, 3D reconstruction, SLAM and so on. In order to obtain the most accurate results, the traditional algorithm needs repeated iterative, so it has a large amount of calculation and time consuming. Most of the existing deep learning algorithms take the left and right images as the input, and obtain the pose parameters based on the semantic features of pixels, so it has a large amount of data and complex model structure. To solve the above problems, a new deep learning network for camera relative pose estimation was proposed, which took the correspondences as the input of network. After obtaining the correspondences between two images, firstly, the correspondences were divided into inliers (the correspondences with small matching error) and outliers (the others correspondences with large matching error) by a classification network. Then, taking all inliers as inputs, the relative rotation and translation parameters of the camera were obtained quickly by a calculation network of camera relative pose parameter. The experimental results show that the proposed method is 1.9 times faster than the traditional algorithm and has less error; The new algorithm has better accuracy precision than the existing deep learning algorithm based on semantic features of pixels, but it processes less data and has a lighter network structure; At the same time, the designed network structure can adapt to the input of different number of correspondences.