Vert.x and Juju
I've been playing with Juju and Vert.x and I got a simple DEVOPS juju charm to deploy a Vert.x app with the minimal effort. Here is the required script:
juju deploy --repository juju-charms local:vertx-app
juju deploy redis-master
juju deploy haproxy
juju add-relation redis-master:db myapp
juju add-relation myapp haproxy
juju add-unit -n 10 myapp
juju expose haproxy
Use the status command to find the public IP address and enjoy! The app at the moment is hard coded to be a Hello World but work is in progress to make it configurable.
vert.x is awsome! Specially with BSON!
Vert.x is a polyglot event-driven application framework that runs on the Java Virtual Machine.
Similar environments written in other programming languages include Node.js for JavaScript that I've been working for the last 1 and a half year.
Vert.x exposes the API currently in Java, JavaScript, Groovy, Ruby and Python.
Scala and Clojure support is on the roadmap.
The application framework includes these features:
- Polyglot. Application components can be written in Java, JavaScript, Groovy, Ruby or Python.
- Simple concurrency model. All code is single threaded, freeing from the hassle of multi-threaded programming.
- Simple, asynchronous programming model for writing truly scalable non-blocking applications.
- Distributed event bus that spans the client and server side. The event bus even penetrates into in-browser JavaScript allowing to create effortless so-called real-time web applications.
- Module system and public module repository, to re-use and share components.
I do like the freedom of being asynchronous and synchronous when needed, I like the real distributed programming model (not like the advertisement from Node.JS which in reallity is a process fork() with modules that require extra coding). With vert.x you just specify the number of instances and you're done!
I disliked the fact that all internal messages were JSON. The event bus is a wonderful component but JSON is too generic. It only allows String, Boolean, Double, Null, Lists and Hashes. Therefore I started a new vert.x module that uses BSON. I picked BSON because it maps easily to JSON and can be a simple replacement.
It turns out you will get a server 33% faster!!! if you use my module!!!
So get it now!!!
ActionScript to Java
One of these days I wanted to convert some AS2/3 to Java. It was obvious that it was simple since ActionScript and Java share lots of language constructs, however it is a tedious task so i created a simple helper application:
package com.jetdrone.as2j;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class AS2J {
private static final Map AS2JTYPES = new HashMap();
public static void main(String[] args) throws Exception {
// e.g.: "~/Flixel2/as/AdamAtomic-flixel-348b55f"
File asSourceDir = new File(args[0]);
String skip = "docs";
// e.g.: "~/src"
File outSourceDir = new File(args[1]);
AS2JTYPES.put("uint", "int");
AS2JTYPES.put("Boolean", "boolean");
AS2JTYPES.put("Number", "float");
AS2JTYPES.put("Array", "java.util.List");
processFile(asSourceDir, asSourceDir.getAbsolutePath(), outSourceDir, skip);
}
private static String getType(String type) {
String t = AS2JTYPES.get(type.trim());
if(t != null) {
return t;
}
return type;
}
private static void processFile(File src, String prefix, File target, String skip)
throws Exception {
for(File f : src.listFiles()) {
if(!f.getName().equals(skip)) {
if(f.isDirectory()) {
processFile(f, prefix, target, skip);
} else {
if(f.getName().endsWith(".as")) {
String fname = f.getAbsolutePath();
String sourceFileName = fname.substring(prefix.length(), fname.length() - 3) + ".java";
convertFile(f, new File(target, sourceFileName));
}
}
}
}
}
private static void convertFile(File src, File dest) throws Exception {
String l;
BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(src)));
PrintWriter out = new PrintWriter(dest);
int i=1;
while((l = in.readLine()) != null) {
try {
l = processPackage(l);
l = processConst(l);
l = processVariable(l);
l = processMethod(l);
l = processConstructor(l);
l = processGetter(l);
l = processSetter(l);
l = processCast(l);
} catch(RuntimeException e) {
System.err.println("ERR (" + i + "): [" + src.getAbsolutePath() + "]");
System.err.println(" --> " + l);
}
out.println(l);
i++;
}
in.close();
out.close();
}
private static final Pattern PACKAGE = Pattern.compile("^\\s*package\\s+(.+)");
private static String processPackage(String line) {
Matcher matcher = PACKAGE.matcher(line);
if (matcher.find()) {
return line.substring(0, matcher.start()) + "package " + matcher.group(1) +
";" + line.substring(matcher.end());
}
return line;
}
// public var exists:Boolean;
private static final Pattern VARIABLE = Pattern.compile("var\\s+(\\w+?)\\s*:\\s*(.+?);");
private static String processVariable(String line) {
Matcher matcher = VARIABLE.matcher(line);
if (matcher.find()) {
if(matcher.group(2).indexOf('=') != -1) {
String type = matcher.group(2).substring(0, matcher.group(2).indexOf('='));
String defaultValue = matcher.group(2).substring(matcher.group(2).indexOf('='));
return line.substring(0, matcher.start()) + getType(type) + " " + matcher.group(1) +
" " + defaultValue + ";" + line.substring(matcher.end());
} else {
return line.substring(0, matcher.start()) + getType(matcher.group(2)) + " " +
matcher.group(1) + ";" + line.substring(matcher.end());
}
}
return line;
}
// static public const A_LIST:uint = 0;
private static final Pattern CONST = Pattern.compile("const\\s+(\\w+?)\\s*:\\s*(\\w+?)\\s*=\\s*(.+);");
private static String processConst(String line) {
Matcher matcher = CONST.matcher(line);
if (matcher.find()) {
return line.substring(0, matcher.start()) + "final " + getType(matcher.group(2)) +
" " + matcher.group(1) + " = " + matcher.group(3) + ";" + line.substring(matcher.end());
}
return line;
}
// public function destroy():void
private static final Pattern METHOD = Pattern.compile("function\\s+(\\w+?)\\s*\\((.*?)\\)\\s*:(\\w+)");
private static String processMethod(String line) {
Matcher matcher = METHOD.matcher(line);
if (matcher.find()) {
return line.substring(0, matcher.start()) + getType(matcher.group(3)) + " " +
matcher.group(1) + "(" + processArgs(matcher.group(2)) + ")" + line.substring(matcher.end());
}
return line;
}
// public function destroy()
private static final Pattern CONSTRUCTOR = Pattern.compile("function\\s+(\\w+?)\\s*\\((.*?)\\)");
private static String processConstructor(String line) {
Matcher matcher = CONSTRUCTOR.matcher(line);
if (matcher.find()) {
return line.substring(0, matcher.start()) + matcher.group(1) + "(" +
processArgs(matcher.group(2)) + ")" + line.substring(matcher.end());
}
return line;
}
private static final String processArgs(String line) {
StringBuffer sb = new StringBuffer();
for(String p : line.split(",")) {
sb.append(processArg(p));
sb.append(", ");
}
if(sb.length() > 0) {
sb.setLength(sb.length() - 2);
}
return sb.toString();
}
// Fixed:Boolean
private static final Pattern PARAM = Pattern.compile("\\s*(\\w+?)\\s*:\\s*(.+)");
private static final String processArg(String line) {
Matcher matcher = PARAM.matcher(line);
if (matcher.find()) {
if(matcher.group(2).indexOf('=') != -1) {
// java has no default values
String defaultValue = matcher.group(2).substring(matcher.group(2).indexOf('=') + 1);
return getType(matcher.group(2).substring(0, matcher.group(2).indexOf('='))) +
" " + matcher.group(1) + " /*" + defaultValue + "*/";
}
return getType(matcher.group(2)) + " " + matcher.group(1);
}
return line;
}
// public get right():Number
private static final Pattern GETTER = Pattern.compile(
"function\\s+get\\s+(\\w+?)\\s*\\(.*\\)\\s*:\\s*(\\w+)");
private static String processGetter(String line) {
Matcher matcher = GETTER.matcher(line);
if (matcher.find()) {
String methodName = matcher.group(1);
return line.substring(0, matcher.start()) + getType(matcher.group(2)) +
" get" + methodName.substring(0, 1).toUpperCase() + methodName.substring(1) +
"()" + line.substring(matcher.end());
}
return line;
}
// public get right():Number
private static final Pattern SETTER = Pattern.compile(
"function\\s+set\\s+(\\w+?)\\s*\\((.*)\\)\\s*:\\s*(\\w+)");
private static String processSetter(String line) {
Matcher matcher = SETTER.matcher(line);
if (matcher.find()) {
String methodName = matcher.group(1);
return line.substring(0, matcher.start()) + getType(matcher.group(3)) +
" set" + methodName.substring(0, 1).toUpperCase() + methodName.substring(1) +
"(" + processArgs(matcher.group(2)) + ")" + line.substring(matcher.end());
}
return line;
}
// var members:Array = (Object as FlxGroup).members;
private static final Pattern CAST = Pattern.compile("\\s*(\\S+?)\\s+as\\s+(\\w+)");
private static String processCast(String line) {
Matcher matcher = CAST.matcher(line);
if (matcher.find()) {
if(matcher.group(1).indexOf('=') != -1) {
String prefix = matcher.group(1).substring(0, matcher.group(1).indexOf('=') + 1);
String value = matcher.group(1).substring(matcher.group(1).indexOf('=') + 1);
return line.substring(0, matcher.start()) + prefix + " (" +
getType(matcher.group(2)) + ")" + value + line.substring(matcher.end());
} else {
if(matcher.group(1).charAt(0) == '(') {
return line.substring(0, matcher.start()) + " ((" + getType(matcher.group(2)) +
") " + matcher.group(1).substring(1) + line.substring(matcher.end());
} else {
return line.substring(0, matcher.start()) + " (" + getType(matcher.group(2)) +
") " + matcher.group(1) + line.substring(matcher.end());
}
}
}
return line;
}
}
New MinPSP plugin 1.0.0
A total refactoring of the plugin is complete, this time the plugin can discover compiler flags and default includes, in order to get it working properly, make sure your minpsp installation is on the path. On windows the installer does this automatically but for other platforms you need to do it yourself.
Once you do this and create a MinPSP lib project/or Makefile project, you will see that default includes get automatically indexed and it can compile out of the box lib projects.
Also a nice addition is that MinPSP debugger gets listed automatically on the debuggers panel.
I will plan to write a new tutorial with the new plugin soon :)
As usual get the plugin from the update site: http://www.jetdrone.com/static/eclipse/update
newlib patches
Lately i have been looking into newlib, although there is again a new release 1.19 i am still looking at 1.18. What is so interesting about it? Well, i have noticed that the pspsdk is using integer math for all it's floating point operations, after studying the documentation is was simple to enable hardware floating point math just by using a configuration flag: --enable-newlib-hw-fp. After this point i've noticed that using the -fast-math with gcc does nothing, why? Apparently newlib only has support (or so it seems to me) for NEC processors, so after a couple of hacking i added basic support for fast math function replacements. I did not add all functions but just some basic ones:
- sinf(float x)
- cosf(float x)
- asinf(float x)
- acosf(float x)
- atanf(float x)
- absf(float x)
- sqrtf(float x)
- expef(float x)
- logef(float x)
The idea here is that just by enabling the -fast-math flag you can benefit from some performance improvement. It is obvious to say that this is all experimental stuff it might work or not with your code, we need to test and see how it behaves. Now, It may be already lots of changes, but something that has been puzzling me for a while is why does ODE (Open Dynamics Engine) work on the PSP, after changing some mails on the ODE mailing list i got some info that NaNs and Infs do not work as expected, so i decided to patch again new lib for that too. Now here is a thing that is more complicated than it sounds. Newlib floating point math has several implementations, ISO, IEEE, X... so which one to change? i opted for the default one, math.h defines 2 functions:
- isnan
- isinf
These were functions that were changed and remained on math.h as macros, that expand to fpclassify comparisons, however there are 2 extra functions:
- __isnanf
- __isinff
These ones are real implementations, so i patched these 2 to use the PSP fpu processor to check if a number is NaN or INF. I will be testing this changes but if some of the users of minpsp, you for example :) would like to help to test just build the toolchain and try it out or let me know and i will build a test toolchain for alpha testers.
MinPSP is comming to Mac OS X
Well, people who know me know that Mac's are really not my thing, i am not a fan of Apple products, i never found my iPod easy to use or even iTunes, they always do what Apple wants, not me... anyway this is no reason for ignore the platform and now with help of a very patient ps2dev community we are getting MinPSP also running on that platform. As of today the same toolchain script can build the binutils, gcc, newlib, sdk, gdb and is stopping on usbpsplink, once we solve this devpaks are next and we are done...
Checking the availability of input on stdin
One of the things MinPSP is looking for, is native port of the homebrew SDK for their respective native platforms, so on Windows we have Windows native executables, on Linux we have Linux native executables, on Linux 64bit... you got the idea... One of the annoying things about this is that not all OSes are POSIX, and not all OSes are from Redmond, so there are quite some quirks that need to be fixed with the crancky #ifdef preprocessor statement. In my last revisit to the MinPSP code I tried to work around the lack of select on file descriptors on windows and converted the pspsh.exe binary from a cygwin binary to a 100% windows native. I isolated all socket code, sockets use select to verify the data availability and for the stdin i assumed that using _kbhit() would be enough. And it has been enough for me since for my tests i always used it from the command prompt, however a long time ago i wrote a small booklet explaining how to plug Eclipse IDE with MinPSP to make it easier to fully develop homebrew applications and there i showed how to invoke pspsh.exe from the eclipse tools. So what was the problem with Eclipse and MinPSP pspsh.exe? Well... i totally forgot that eclipse will just start the process and communicate with it via pipes!!! When you have pipes the _kbhit() function will never return a value other than 0. Having this in mind pspsh would never read anything from stdin since all I/O in the application is non blocking. So after long time of googling and testing i found this piece of code:
static int is_pipe = 0;
static HANDLE input_handle = 0;
int input_available()
{
DWORD nchars;
/* When using Standard C input functions, also check if there
is anything in the buffer. After a call to such functions,
the input waiting in the pipe will be copied to the buffer,
and the call to PeekNamedPipe can indicate no input available.
Setting stdin to unbuffered was not enough, IIRC */
if (stdin->_cnt > 0)
return 1;
if (is_pipe)
{
/* When running under a GUI, you will end here. */
if (!PeekNamedPipe(input_handle, NULL, 0, NULL, &nchars, NULL))
/* Something went wrong. Probably the parent program exited.
Could call exit() here. Returning 1 will make the next call
to the input function return EOF, where this should be
catched then. */
return 1;
return (nchars != 0);
}
else
return _kbhit() != 0; /* In "text-mode" without GUI */
}
Amazingly this code works for me, and lets me pool stdin from pipe for data available to be read. The only thing missing here is how is the is_pipe variable initialized. I've found 2 ways one using the function _isatty( _fileno(stdin)) or using Win32 API, I've chosen this way:
DWORD dw; input_handle = GetStdHandle(STD_INPUT_HANDLE); is_pipe = !GetConsoleMode(input_handle, &dw);
And that was it, now you can handle non blocking input from console and from pipes!!! Hurray!!!
Experimenting with OpenCL and Java
MinPSPW has gone to Mars
Long time ago it was possible to write homebrew code for the PlayStation Portable using the D Programming Language, however with the upgrade of GCC to the 4.1, 4.2 and later on to 4.3 the patch didn't work anymore. During the last month I've been studying the GDC bitbucket project and trying to port it back to the MinPSPW toolchain. Although it seemed trivial, because the new GDC project has updated the patch for GCC 4.3 it proven not practical for the cross toolchain. After patching a lot of stuff I finally have a build for now only on Linux that can compile and link a simple Hello World example.
So for now on it will be possible to write code such as:
// imports
// extern (C) allows D to call C functions directly, no need to wrap
extern (C) void pspDebugScreenInit();
extern (C) void pspDebugScreenPrintf(char*,...);
extern (C) int SetupCallbacks();
extern (C) int sceKernelSleepThread();
import std.string;
class HelloPsp
{
public:
void sayHello()
{
pspDebugScreenPrintf(toStringz(m_msg));
}
char[] m_msg = "Hello D on Psp\n";
}
int main()
{
SetupCallbacks();
pspDebugScreenInit();
auto hello = new HelloPsp();
hello.sayHello();
sceKernelSleepThread();
return 0;
}
So why D? D is a systems programming language. Its focus is on combining the power and high performance of C and C++ with the programmer productivity of modern languages like Ruby and Python. Special attention is given to the needs of quality assurance, documentation, management, portability and reliability.
The D language is statically typed and compiles directly to machine code. It's multiparadigm, supporting many programming styles: imperative, object oriented, and metaprogramming. It's a member of the C syntax family, and its appearance is very similar to that of C++.
I am planning to update the toolchain and start testing with Windows to be released once the new Ubuntu version is out.
Performance performance performance!!!
Fast 2D render
In the last couple of weeks I started a research activity around map rendering. I know that there is plenty of map renderers, and that Mapnik is pretty good, however I wanted to experiment so I need something simple yet fast. This lead me to research on 2D rendering libraries. Libraries that are free since I have to budget and portable since I work mostly with Linux and sometimes with Windows.
After googling for a while I found 2 big libraries:
- Cairo Graphics
- AntiGrain Geometry
The first one is C and the second C++ of course I'd go with the first, I do prefer C over C++ but that could be a topic for other post. These libraries are pretty impressive but I wanted something portable and that is when I have found OpenVG. The concepts behind OpenVG made me think, WOW I want it, so I started a quest for a implementation that was not software based... and I couldn't find one... at least for free.
Facing this setback I decided to research how does java and the jvm fix this problem. Having seen the latest versions of Netbeans they must have some optimized code to make all that swing code fast, so I started looking into Java2D. Searching the web I found lots of opinion pages saying that Java is slow, Java2D is slow but still I wanted to give it a try. After going over the 2D tutorial I found out that in fact Java2D can benefit from hardware acceleration. On Windows acceleration is achieved through DirectDraw and on Linux through OpenGL. According to the tutorial one can request accelerated rendering operations by hinting the jvm that our target image should be accelerated like this:
GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment(); GraphicsConfiguration gc = ge.getDefaultScreenDevice().getDefaultConfiguration(); BufferedImage img = gc.createCompatibleImage(width, height, Transparency.TRANSLUCENT); img.setAccelerationPriority(1);
We create a BufferedImage that matches our graphic configuration so the chances that we end up this image on VRAM are bigger and then hint the JVM that it should accelerate it. This by itself already brings performance improvement but if we really want fast images there is another kind of image, the VolatileImage. This kind of image is saved in a OpenGL pbuffer and on windows it can be really volatile. This means that even though you are drawing into it it may end up clobbered by other application that needed to write into the VRAM. This kind of image is really fast but has the inconvenient that it cannot be used directly with the ImageIO class to write to disk, you still need to create a BufferedImage to write to and then you can save.
GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment(); GraphicsConfiguration gc = ge.getDefaultScreenDevice().getDefaultConfiguration(); VolatileImage vImage = gc.createCompatibleVolatileImage(width, height, Transparency.TRANSLUCENT);
It is up to you how fast you need your images to be...