Are you ready to take organization of your children's party to next level? Wouldn't it be wonderful if you would have some robots do all your work, so you can relax, enjoy and hangout with the guests? Maybe we are not far from it but we'll wait for a few years for something like this to happen. Except if we talk about robot-photographer. Ok, not exactly real robot, more automatic photographing from ordinary cameras, webcams or mobile phones.
We can use simple OpenCV scripts to take photos while streaming. It can be automatically without the need for our labor. We can choose the moment of shooting and there are several options. Cameras can be set to take photos after some time (for example, every 50 seconds), when some face is detected, when there are no movement or when some shape or text is detected. Also, we can use different types of streaming, but those streams have to be connected with computers running OpenCV. I'll describe each type and share link to github code, so you can use any type you like or is best suited for you. And of course, any suggestions for improvements are more than welcome.
I have to inform you that all this will be to hard for beginners in Computer Vision. But I will keep doing tutorials and everyone will be able to understand this eventually.
The code will always start with the importing the libraries.
import numpy as np import cv2 import time
First, I'll describe how you can stream with the different types of cameras. We'll use streams from built-in cameras, external cameras or streaming through internet. The most simple way is laptop with the webcam, whether it's built-in or external connected to laptop. Also you can use DSLR camera with little bit of preparations. The code to use stream looks like this:
cap = cv2.VideoCapture(0)
The other way is to stream to internet and then use that internet stream. If you use your mobile phone, useful applications would be IP webcam. You can also use IP cameras and the code to get internet stream is different:
cap = cv2.VideoCapture('http://YOUR_URL/video')
Now, we are starting with the code. First method we'll use is going to be the easiest one. It will take 100 photos during stream every few seconds. Of course, you can put another number of photos if you like.
def stream(i):
while cap.isOpened():
ret, frame = cap.read()
cv2.imshow("frame", frame)
cv2.imwrite('image%s.jpg' % i, frame)
time.sleep(3)
break
for x in range(100):
stream(x)
The whole code can be found here.
If that's to simple for you and you want to choose better moment for capturing, next method will be much better. We can set our stream for taking photos when someone's face is detected. To do that, we'll need haarcascade classifier.
face_cascade = cv2.CascadeClassifier('haar_cascades.xml')
Then, we need to make a function which will capture images during stream. First, we'll initiate an infinite loop (to be broken later by a break statement), where we have ret and frame being defined as the cap.read(). Then, we'll convert those frames into grayscale frames for better detection.
def stream(i): while cap.isOpened(): ret, frame = cap.read() gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
Now, we need to call detectMultiscale() function and choose arguments. I think it's better to try to avoid false detections as much as we can so I'll put minNeighbors parameter to 1.3 and minSize parameter to 10.
faces = face_cascade.detectMultiScale(gray, 1.3, 10)
And finally, it's time for capturing an image when face is detected.
if len(faces) > 0: cv2.imwrite('image%s.jpg' % i, frame) time.sleep(3) break
We can call stream() function as much time as we want. I'll put 100 times, but you can any number you want. And if you want to make sure it will shoot the whole time, just put some really big number.
for x in range(100): stream(x)
The code can be found here.
Above 2 methods work fine, but some of the images may be blurry because of the motion. Because of this reason, great method would be to take photo when there are no motion. Here, we'll use 2 frames of stream and their differences.
ret, frame1 = cap.read()
ret, frame2 = cap.read()
diff = cv2.absdiff(frame1, frame2)
We will do a little processing, convert it to grayscale, blur, determine the thresholds, remove the noise and find contours at the end.
gray = cv2.cvtColor (diff, cv2.COLOR_BGR2GRAY) blur = cv2.GaussianBlur(gray, (5,5), 0) _, thresh = cv2.threshold(blur, 20, 255, cv2.THRESH_BINARY) dilated = cv2.dilate(thresh, None, iterations=3)
contours, _ = cv2.findContours(dilated, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
Next, we have to make sure contour area is less than 100 for 2 seconds. This way, there shouldn't be any motion. After that, we can start a new stream to capture next photo.
for contour in contours:
time.sleep(2)
array = []
if cv2.contourArea(contour) < 100:
no_motion= True
else:
no_motion=False
array.append(no_motion)
print(all(array))
if (all(array)==True):
cv2.imwrite('image%s.jpg' % i, frame1)
cap.release()
Look at the code here.
I know everyone would like more photos of the children, so next 2 methods are going to be about that. We need something that will make children recognizable and my idea is party hat. We can write applications that can recognize hat itself or something on the hat, for example some text or balloons which can be recognized as circles.
I tried to find satisfying way for cone of the party hat to be recognizable, but I failed for now and I hope some of you will help me. So, we'll continue will circle detection. First, we'll do a little bit of processing.
while cap.isOpened(): _, img = cap.read() gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) blur = cv2.medianBlur(gray, 5)
Now, we can use cv2.HoughCircles() function, which takes 8 arguments. As there are many circles on our hats, we can be more strict in our parameters. I put param1 to 200 and param2 to 60 to avoid false detections, but I suggest to check the best possible parameters with your hats. Check
circles = cv2.HoughCircles(blur, cv2.HOUGH_GRADIENT, 1, 20, param1=200, param2=60, minRadius=0, maxRadius=0)
Check the whole code.
If you want reliable algorithm for text recognition, don't put something to long and make sure text is clear to view. Let's see why...
Let's check canny edges.
You can see it's even hard for us to read the text, we need hats with simple text.
Text for my daughter's birthday will be "M" or "3". M is first letter of her name Marija and it's going to be her third birthday. I suggest you use something similar to make things simple.
We'll use PIL and pytesseract libraries.
from PIL import Image import pytesseract
Our function will look a little bit different, we'll use pytesseract.image_to_string() function that extracts text from an image.
def stream(i): while cap.isOpened(): ret, frame = cap.read() text = pytesseract.image_to_string(frame) if (text=="M"): cv2.imwrite('image%s.jpg' % i, frame) break
Have a look at the whole code.
Now, we're done with ground photos, let's take some from air. Autonomously, of course. We just need a drone, so we'll use my home-made one. It's a quadcopter with Pixhawk autopilot, controlled by Raspberry Pi 4 and uses RPI HQ camera.
After hardver, let's present our softver. Drone is run by ROS and camera by python-picamera. We'll make 3 scripts and you can choose one of them (except you have 3 drones to run all of them). First one will save a video, second one will take a photos like those ones before and the third will be combination of these two.
First, we'll import the libraries.
from __future__ import print_function import os import numpy as np import math import sys import rospy import picamera import mavros import threading import time import mavros_msgs from mavros import command from std_msgs.msg import String from mavros_msgs import srv from geometry_msgs.msg import Twist, TwistStamped from mavros_msgs.srv import SetMode, CommandTOL, CommandBool from mavros_msgs.msg import State from mavros_msgs.srv import *
Next, we'll make class Drone with functions for connecting, arming, takeoff, movement, rotation and video recording and landing at the end.
class Drone:
def __init__(self): self.rate = 1 self.connected = False
def connect(self, node: str, rate: int): rospy.init_node(node, anonymous=True) self.rate = rospy.Rate(rate) self.connected = True rospy.loginfo("Connected...") def arm(self): print("Arming...") rospy.wait_for_service('/mavros/cmd/arming') try: armService = rospy.ServiceProxy('/mavros/cmd/arming', mavros_msgs.srv.CommandBool) armService(True) except rospy.ServiceException as e: print ("Service arm call failed: %s" %e) def takeoff(self): print("Takeoff ...") rospy.wait_for_service('/mavros/cmd/takeoff') try: takeoffService = rospy.ServiceProxy('/mavros/cmd/takeoff', CommandTOL) response = takeoffService(altitude = 2, latitude = 0, longitude = 0, min_pitch = 0, yaw = 0) rospy.loginfo(response) except rospy.ServiceException as e: print ("Service takeoff call failed: %s"%e) def move(self, x, y): pub = rospy.Publisher('/mavros/setpoint_velocity/cmd_vel', TwistStamped, queue_size=10) vel = TwistStamped() vel.twist.linear.x= x; vel.twist.linear.y= y; pub.publish(vel); time.sleep(5) def rotate(self,z): vel = TwistStamped() vel.twist.rotate.z = z time.sleep(600) def land(self): print("Landing... ") rospy.wait_for_service('/mavros/cmd/land') try: landService = rospy.ServiceProxy('/mavros/cmd/land', CommandTOL) response = landService(altitude = 0, latitude = 0, longitude = 0, min_pitch = 0, yaw = 0) rospy.loginfo(response) except rospy.ServiceException as e: print ("service land call failed: %s. The vehicle cannot land "%e)
In the main part, we'll call these functions. It is better to start few meters from the point of recording so we use move() function before rotating around and recording and after that. You can also change it to fly around a little bit if you want different kind of video.
def main(args): v=Drone() v.connect("drone",rate=10) time.sleep(3) rospy.wait_for_service('/mavros/set_mode') change_mode = rospy.ServiceProxy('/mavros/set_mode', SetMode) response = change_mode(custom_mode="GUIDED") print("Mode set to GUIDED") time.sleep(3) v.arm() time.sleep(3) v.takeoff() time.sleep(3) camera = picamera.PiCamera() camera.resolution = (640, 480) camera.start_recording('my_video.h264') camera.wait_recording(650) vel = [[0, 0], [-5, 0], [0, -5]] i = 0 while i < len (vel): x = vel [i] [0] y = vel [i] [1] v.move(x, y) i = i+1 v.rotate(0.2) vel = [[0, 0], [0, 5], [5, 0]] i = 0 while i < len (vel): x = vel [i] [0] y = vel [i] [1] v.move(x, y) i = i+1 camera.stop_recording() v.land() time.sleep(10) if __name__ == "__main__": main(sys.argv)
Check the code here.
If we want to use drone for more photos, code would look like this.
from picamera.array import PiPGBArray from picamera import PiCamera import time import random camera = PiCamera() camera.resolution = (640, 480) camera.framerate = 32 rawCapture = PiRGBArray(camera, size=(640, 480)) for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True): image = frame.array rawCapture.truncate(0) for i in range(50): camera.start_preview() time.sleep(2) camera.capture('image%s.jpg' %i) time.sleep(5) break
The code is here.
And, of course, combination is also always possible. The code is here and you can change duration of video or take more photos. It's up to you.
I hope you enjoyed reading this post and I'm looking forward to hear your comments and experiences. And the most important, please don't hesitate with any kind of suggestion for improvements.