Using OpenCV on iPhone
OpenCV is a library of computer vision developed by Intel, we can easily detect faces using this library for example. I’d note how to use it with iPhone SDK, including the building scripts and a demo application. Here I attached screen shots from the demo applications.
Support OpenCV 2.0.0 and iPhone SDK 3.x
I updated the script and made patches for OpenCV 2.0.0 and iPhone SDK 3.x! (Updated 11/15/2009)
Getting Started
All source codes and resources are opened and you can get them from my github repository. It includes pre-compiled OpenCV libraries and headers so that you can easily start to test it. If you already have git command, just clone whole repository from github. If not, just take it by zip or tar from download link on github and inflate it.
% git clone git://github.com/niw/iphone_opencv_test.git
After getting source codes, open OpenCVTest.xcodeproj with Xcode, then build it. You will get a demo application on both iPhone Simulator and iPhone device.
Building OpenCV library from source code
You can also make OpenCV library from source code using cross environment compile with gcc. I added some support script so that you can easy to do so. The important point is that iPhone SDK doesn’t support dynamic link like “.framework”. We have to make it as static link library and link it to your application statically.
Getting source code from sourceforge. I tested with OpenCV-2.0.0.tar.bz2.
Extract downloaded archive on the top of project project directory
% tar xjvf OpenCV-2.0.0.tar.bz2Apply patch for iPhone SDK
% cd OpenCV-2.0.0 % patch -p0 < ../cvcalibration.cpp.patch_opencv-2.0.0Following next steps to build OpenCV static library for simulator. All files are installed into
opencv_simulatordirectory.% cd OpenCV-2.0.0 % mkdir build_simulator % cd build_simulator % ../../configure_opencv % make % make installFollowing next steps to build OpenCV static library for device All files are installed into
opencv_devicedirectory.% cd OpenCV-2.0.0 % mkdir build_device % cd build_device % ARCH=device ../../configure_opencv % make % make install
Patch and Configure support script
OpenCV 2.0.0 (and also 1.1.0) has a glitch which refuse builing the library with iPhone SDK.
You need to apply the patch cvcalibration.cpp.patch so that you can build it.
Congiure support script configure_opencv has some options to build OpenCV with iPhone SDK.
Try to --help option to get the all options of it.
Converting images between UIImage and IplImage
OpenCV is using IplImage structure for processing, and iPhone SDK using UIImage object to display it on the screen. This means, we need a converter between UIImage and IplImage. Thankfully, we can do with iPhone SDK APIs.
Creating IplImage from UIImage is like this.
// NOTE you SHOULD cvReleaseImage() for the return value when end of the code.
- (IplImage *)CreateIplImageFromUIImage:(UIImage *)image {
// Getting CGImage from UIImage
CGImageRef imageRef = image.CGImage;
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// Creating temporal IplImage for drawing
IplImage *iplimage = cvCreateImage(
cvSize(image.size.width,image.size.height), IPL_DEPTH_8U, 4
);
// Creating CGContext for temporal IplImage
CGContextRef contextRef = CGBitmapContextCreate(
iplimage->imageData, iplimage->width, iplimage->height,
iplimage->depth, iplimage->widthStep,
colorSpace, kCGImageAlphaPremultipliedLast|kCGBitmapByteOrderDefault
);
// Drawing CGImage to CGContext
CGContextDrawImage(
contextRef,
CGRectMake(0, 0, image.size.width, image.size.height),
imageRef
);
CGContextRelease(contextRef);
CGColorSpaceRelease(colorSpace);
// Creating result IplImage
IplImage *ret = cvCreateImage(cvGetSize(iplimage), IPL_DEPTH_8U, 3);
cvCvtColor(iplimage, ret, CV_RGBA2BGR);
cvReleaseImage(&iplimage);
return ret;
}
Don’t forget release IplImage after using it by cvReleaseImage!
And creating UIImage from IplImage is like this.
// NOTE You should convert color mode as RGB before passing to this function
- (UIImage *)UIImageFromIplImage:(IplImage *)image {
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// Allocating the buffer for CGImage
NSData *data =
[NSData dataWithBytes:image->imageData length:image->imageSize];
CGDataProviderRef provider =
CGDataProviderCreateWithCFData((CFDataRef)data);
// Creating CGImage from chunk of IplImage
CGImageRef imageRef = CGImageCreate(
image->width, image->height,
image->depth, image->depth * image->nChannels, image->widthStep,
colorSpace, kCGImageAlphaNone|kCGBitmapByteOrderDefault,
provider, NULL, false, kCGRenderingIntentDefault
);
// Getting UIImage from CGImage
UIImage *ret = [UIImage imageWithCGImage:imageRef];
CGImageRelease(imageRef);
CGDataProviderRelease(provider);
CGColorSpaceRelease(colorSpace);
return ret;
}
Ok, now you can enjoy with OpenCV with iPhone!
Frequently Asked Questions
I can’t build and run this demo application for iPhone device, I can build and run it on iPhone simulator though… why? I got next error when building.
ld warning: in /usr/local/lib/libcv.dylib, file is not of required architecture ld warning: in /usr/local/lib/libcxcore.dylib, file is not of required architecture Undefined symbols: "_cvCreateMemStorage", referenced from: -[OpenCVTestViewController opencvFaceDetect:] in OpenCVTestViewController.o "_cvGetSeqElem", referenced from: -[OpenCVTestViewController opencvFaceDetect:] in ......- Have you ever installed OpenCV for MacOS X? This error is because of the linker using wrong library for MacOS X instead of iPhone device. I solved this problem so that you can now build it for iPhone device. Please git pull or download the package again from github.
Change Log
- 12/21/2009 - Support Snow Leopard + iPhone SDK 3.1.2, Thank you Hyon!
- 11/15/2009 - Support OpenCV to 2.0.0 + iPhone SDK 3.x
- 03/14/2009 - Release this project with OpenCV 1.0.0 + iPhone SDK 2.x
One more thing…
I mentioned that the face detection using OpenCV takes very long time. For example detecting with iPhone screen size image, it takes 10 seconds or more…hmmmm
License
This sample is under MIT License.
Good!
Same paper here:
http://www.computer-vision-software.com/blog/2009/04/opencv-vs-apple-iphone/comment-page-1/#comment-106
この情報があると大変助かります、ありがとうございます！。opencvをダウンロードし解凍するとエラーがでますので、恐らく違うopencvのファイルだと思います。
opencv-1.1pre1.tar.gz の具体的なリンクを教えていただけないでしょうか？
> イグナチオ さん
数日前のSourceforgeの改変で、ダウンロードリンクが変更になっており、Safariではダウンロードできなくなっています。opendv-1.1pre1.tar.gzはopencv-linuxの下にありますが、Firefoxを使うなどしてダウンロードしてください。
Hi, Great stuff here. I was wondering if you got openCV's video capture api to work?
It would be great if you share your sample code
Whoops! You did ;-) I didn't realize the bottom link.
> Keo san
Hi, unfortunately OpenCV stuff doesn't support any iPhone specific functions includes Camera operations so far.
> Dongpyo san
Right! You can grab the sample from my github repository!
助かったよ！
Thanks for providing an example project for this! This is a big help.
I got the armv6 to compile, but not the sim. anyone have any trouble with that? the error i get is ** No targets specified and no makefile found. Stop.
thanks.
I think i figured it out. for the sim stuff, you need to make a few changes. see the 4 lines below. I'm running snow leopard + sdk3.1. Snow leopard requires you to change from darwin9 to darwin10. sdk3.1 requires you to change the sdk appropriately (note you don't need this change for arm, since you can still build to target 2.2.1 - minor change from 2.2 to 2.2.1 in the armv6.sh file). Hope that helps folks.
CC=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/i686-apple-darwin10-gcc-4.0.1 \
CXX=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/i686-apple-darwin10-g++-4.0.1 \
CFLAGS="-arch i386 -isysroot /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.sdk" \
CXXFLAGS="-arch i386 -isysroot /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.sdk" \
jeff, please, where I have to change this?
CC=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/i686-apple-darwin10-gcc-4.0.1 \
CXX=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/i686-apple-darwin10-g++-4.0.1 \
CFLAGS="-arch i386 -isysroot /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.sdk" \
CXXFLAGS="-arch i386 -isysroot /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.sdk" \
> Jeff, asffa
Thanks, I know current configure file doesn't work with iPhone SDK 3.x (and on Snow Leopard?) Unfortunately I'm quite busy for now, I then can't address this issue.. though it might be helpful! Thanks!
Anyone got a solution?! Yoshimasa please, could you help us?! how can we get it work on 3.1 sdk?
Hey, If you use Yoshimasa-san 's library you don't need to compile a new opencv library.
I would like to know how can I compile opencv 2.0 for iphone?
I have tried this (based in README file form Yoshimasa-san project) But no success.
% cd OpenCV-2.0.0
% mkdir build_armv6
% pushd build_armv6
% ../configure_armv6.sh
% make
% make install
It is failing just after I try to run the configure_armv6.sh script.
It start checking things and ...
checking for C++ compiler default output file name...
configure: error: in `/Users/nacho4d/Downloads/OpenCV-2.0.0/build_armv6':
configure: error: C++ compiler cannot create executables
>Jeff
Can you advice on what changes you did for compiling for the device (build_armv6)
I am trying this but did not work.
#!/bin/sh
PREFIX=`pwd`/`dirname $0`/../opencv_armv6
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin
../configure --prefix=${PREFIX} \
--host=arm-apple-darwin \
--enable-static \
--disable-shared \
--without-python \
--without-ffmpeg \
--without-1394libs \
--without-v4l \
--without-imageio \
--without-quicktime \
--without-carbon \
--without-gtk \
--without-gthread \
--disable-apps \
CC=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/arm-apple-darwin9-gcc-4.2.1 \
CXX=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/arm-apple-darwin9-g++-4.2.1 \
CFLAGS="-arch armv6 -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.1.sdk" \
CXXFLAGS="-arch armv6 -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.1.sdk" \
CPP=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/cpp \
CXXCPP=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/cpp \
AR=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/ar
I got
checking for C++ compiler default output file name...
configure: error: in `/Users/nacho4d/Downloads/OpenCV-2.0.0/build_armv6':
configure: error: C++ compiler cannot create executables
Hi there, I'm updating this project for iPhone OS 3.x and latest SDK with OpenCV 2.0. I'll post new article it, thank you!
Thank you very much! It's great for me ^_^
Hi Yoshimasa,
After installing success for target simulator, I got an error message when 'make install' with the target device:
ARCH=device ../configure_opencv GCC_VERISON=4.0
make install-exec-hook
ldconfig
make[3]: ldconfig: Command not found
make[3]: [install-exec-hook] Error 127 (ignored)
test -z "/Users/mac/CodeTemplates/iPhoneVideoPerformance/05.OpenCV/Source/OpenCV-2.0.0/build_device/../opencv_device/lib/pkgconfig" || ../autotools/install-sh -c -d "/Users/mac/CodeTemplates/iPhoneVideoPerformance/05.OpenCV/Source/OpenCV-2.0.0/build_device/../opencv_device/lib/pkgconfig"
/usr/bin/install -c -m 644 opencv.pc '/Users/mac/CodeTemplates/iPhoneVideoPerformance/05.OpenCV/Source/OpenCV-2.0.0/build_device/../opencv_device/lib/pkgconfig'
> doungtu
Hi, As you see on this error message, just ignore it.
We're not using ldconfig because it is for the dynamic linking of OpenCV, we're using static link because of the iPhone limitation.
素晴らしいですね。参考にさせてもらいました。こういった情報があると助かります。どうもありがとうございます！
Hello Yoshima!
First thank you sharing your great work because your tutorial among others was most helpful!
I have question about this comment with method for converting to IplImage (UIImageFromIplImage) when it says: "NOTE You should convert color mode as RGB before passing to this function".
In my project im converting image first to IplImage, then doing some cv pixel changes and then converting it back to UIImage and show it back on imageView. Problem occurs with colors on that picture (it becomes blue). Like there was some problem with color chanels.
Regarding to comment about changing RGB color mode is there something I need to call before converting back to UIImage?
tnx for your reply!
I found out that color mode is inverted so its BGR.
But when I try to convert it back to RGB it doesnt work :(
Okey I manage to figure out the problem. I had to call the same convertion as in "CreateIplImageFromUIImage" method.
"cvCvtColor(image, image2, CV_RGBA2BGR);".
BTW why do we need to transform colors at all?
> Miha
Hi, Miha. Yes, you can use cvCvtColor() because we're getting UIImage from CGImage and CGImageCreate requires the RGBA or ARGB formatted byte array. see "Color Spaces and Bitmap Layout" section of "Quartz 2D Programming Guide"
Tnx, it makes sense now!
In my project im doing some body detection and i need to allocate edge (figure) of body.
Do you maybe know how can i get position of body (figure)?
Is it possible to do anything in realtime yet ? I don't mean the CPU or RAM can handle it, I mean does the OS let you do this ?
Thanks for your great article, Yoshimasa.
Hi, Thanks for your the pre-compiled OpenCV 2.0.0.
When I try to build your demo app I get an error saying the following files are missing:-
cxoperations.hpp, cxmat.hpp, cxflann.h.
Thank you
> Miha
You have to make some training data for detecting specific objects by Open CV. Try to google "haar training".
> Parsa
I think somebody is making a patch to increase the speed to detect on iPhone and I hope it works in realtime.
> Jason
Weird... Could you try to clone from github to another directory then build it?
Has anyone tried using this with live video and reference frames instead of still images?
GREAT! thank you so much for this porting!
Little question: how would you do to port OpenSurf to iPhone??
I think it would be nice to see it in action on the device since the OpenSurf output is slightly different from the OpenCv one..
Best
link to OpenSurf: http://code.google.com/p/opensurf1/
First off great work this really helps. Second has anyone run into the problem that the configure file fails to execute "fails sanity check?". The log file says that there was an error running conftest.cpp "bad value (core2) for -mtune= switch". I don't know why this would be a bad value, I am on an Intel Xeon which is a dual core machine. Thanks in advance for the help.
jeff said at 09/23, 2009
I got the armv6 to compile, but not the sim. anyone have any trouble with that? the error i get is ** No targets specified and no makefile found. Stop.
Hi, I got the same problem when I tried to do the "make" here:
% cd OpenCV-2.0.0
% mkdir build_simulator
% ../../configure_opencv
% make
% make install
Has anyone know how to solve this problem? (and I didn't see the Makefile in the downloaded codes......)
Thanks!
I need a different variant than the ones discussed above: opencv1.1+SDK 3.x.
I successfully built the opencv2.0+SDK 3.x project, but with opencv1.1 build, I made the following changes to 'configure_opencv', similar to 'jeff' above:
CC=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/i686-apple-darwin10-gcc-4.2 \
CXX=/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin/i686-apple-darwin10-g++-4.2 \
CFLAGS="-arch i386 -isysroot /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.sdk" \
CXXFLAGS="-arch i386 -isysroot /Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.1.sdk" \
This allows me to 'make' the simulator build, but fails during the subsequent 'make install'
Can anyone confirm that it is possible to build opencv1.1 with SDK 3.x?
Thanks!
Just a simple note for maplevincent:
I think the steps laid out by Yoshimasa-san omit a 'cd' instruction, which I found was necessary to make the project. The added line is below:
% cd OpenCV-2.0.0
% mkdir build_simulator
--> % cd build_simulator
% ../../configure_opencv
% make
% make install
Hi, Kevin, and All users:
Thank you for your comment! Yes, it is my mistake. I'd update the article in English(it is correct in Japanese).
And for Snow Leopard users, current script doesn't work on Snow Leopard. I'd also fix it soon!
Hi John,
I fixed the issue you pointed out at that comment. Please use the latest source at github repository!
Hi All,
I changed my typo and mistakes on the article and update the source code to support Snow Leopard. If you have some trouble on Snow Leopard when configuring OpenCV on it, please update the script from github.
Thank you!
Hello, Yoshimasa,
Thanks for updating your source, and marking that small change I noted. There's another small error in your instructions for the device build:
% ARCH=device ../../configure_opencv
should be:
% ../../configure_opencv ARCH=device
as below:
% cd OpenCV-2.0.0
% mkdir build_device
% cd build_device
% ../../configure_opencv ARCH=device
% make
% make install
Thanks again for the new project updates, but I'm still unable to build with opencv1.1+SDK 3.x. on Snow Leopard.
I have a feeling that it has to do with:
ARCH_HOST=i686-apple-darwin9
But changing to 'ARCH_HOST=i686-apple-darwin10' doesn't seem to help.
Thanks,
-Kevin
Hi Giacomo,
I tested on iPhone then OpenSURF works on my iPhone 3G though, it is super slow...
Hi Kevin,
My current configure helper may not support OpenCV 1.1. The older configure script which you can fetch from my github repository may help your problem.
Hello!
I could successfully compile the libraries, and even got a step further and compiled them into a universal lib.
My question is, how can I start a project from scratch using OpenCV? I looked into the sample project you provide but can't find any pointers to where to configure OpenCV. I find myself with this errors after compiling:
Undefined symbols:
"_cvMinAreaRect2", referenced from:
_cvMinAreaRect in HelloOpenCVAppDelegate.o
I think the are due to the library not being found, which leads me to the question: how to configure opencv properly in xcode?
Thanks in advance!
Thanks again for such a fast reply, I will have a look for your earlier github entries for a configuration script which can work with openCV 1.1pre.
I realize that openCV 1.1pre is a little old, and would rather build around 2.0, of course. If I'm lucky, I will be able to.
Thanks again,
-Kevin
Hello, Yoshimasa,
Thanks for your nice instructions about compiling opencv libs.
The opencv libs I build are working good on both device and simulator, but there are 7 warnings like the one below.
ld: warning: can't add line info to anonymous symbol __ZN2cv9ExceptionD1Ev.lsda from ./opencv_simulator/lib/libcxcore.a(lib_cxcore_la-cxmathfuncs.o)
Is there any way I can do to fix this kind of warning?
p.s. I am using Snow leopard, and the latest version of script from github
Thanks in advance!
Hi Kenny, You can ignore any warnings.
Thanks for your fast reply!
This article really help me a lot
とても助かりました。ありがとうございました。
HighguiのVideo writer関連が使えるかな、と思って試してみましたが、やっぱり無理でした。
> Hamken100perさん
そうですねー、そのあたりすこし頑張ればできると思っていますが、がんばらないとダメですね。そもそもAPIがまだ公式ではないので、なんとも言い難いところがありますが。
So great example !
Is that legal to publish application to appStore using opencv library in iPhone ?
There are some camera applications in appStore now, does that can use opencv library now ? what sort of pop framework used there ?
A really awesome project !
Are there any published apps from app store which is based on opencv ?
How about the performance ? Is that very slow speed ?
welcome further communication
forrest.shi < at > gmail < dot > com
> Forrest
Yes, you can use it on AppStore. Before asking here like you've commented on, I think you had better try this demo out.
I am a newbie to Mac world.
I would like to ask where should I extract the tar file? After extract, and run the patch command, it says there is no such directory.
(Maybe this is a stupid question :P )
> Kenny
Hi, you should get this demo project from github then move its directry then extract OpenCV there and move into OpenCV directory then run the patch command though, I think you don't need to apply patch to OpenCV. you can use pre-compiled binaries of OpenCV I included into this demo project. Just open the project with XCode.
Hi Yoshimasa, your demo is awesome, really!!
I tried to build the files from source for the simulator and i get the following at the end:
configure: error: in `/Users/jasan/Apps/Test1/OpenCV-2.0.0/build_simulator':
configure: error: C++ compiler cannot create executables
is there anything i am missing? thanks
PS: I checked the config.log and everything is ok, except for:
conftest.cpp:11:19: error: no include path in wich to search for stdio.h
...
after that it gives error in some I/O operations
i know this has nothing to do with your script, but if someone have any clue please help me. thanks.
Very good example it is really helpful for understanding
Yoshimasa,
Thanks much. I have downloaded and tested the demo. That's great. I am now learning what the code is doing :P
If I write my own apps with your lib. Is that OK?