The Fourth Industrial Revolution, or Industry 4.0, has among its goals improving efficiency and increasing throughput. To achieve these goals, the smart factory for Industry 4.0 requires digital techniques of artificial intelligence (AI), image signal processing (ISP), and advanced robotics that can support the system integration of software and hardware. An interactive mode of human–robot collaboration (HRC) is a significant focus of the smart factory to increase the intelligence of robots and further promote autonomous systems. In this study, we have developed a scheme for object inspection based on HRC. Different from automatic optical inspection (AOI) without the involvement of an operator, the inspection scheme developed significantly increases interactions between the operator and the manipulator. In the proposed HRC-based object inspection, the manipulator is viewed as an assistant to the operator to deliver the inspected object by grabbing, transporting, and releasing it. When the manipulator moves the object to the side of the operator, the operator then makes a detailed inspection on the basis of his expert knowledge; after completing the inspection, the operator executes consecutive hand poses to “tell” an image sensor located in the manipulator the results of the inspection, including both the object quality rank and the object material type. Hand pose recognition is used to classify the hand poses of the operator. In this study, a two-phase hand pose recognition strategy was adopted on the basis of simple finger segmentations. A robust and fast recognition approach that is capable of making classifications of hand poses of numeric symbols combined with non-numeric expressions was developed. The HRC-based object inspection was carried out in a laboratory environment that simulated a real system. Experiments on HRC-based object inspection with classifications of five hand poses of non-numeric expressions and nine hand poses of numeric symbols demonstrated both the effectiveness and efficiency of this approach.