Pages

Monday, November 11, 2019

MPLAB X IDE - Linux vs Windows pre-build scripts

So MPLAB X IDE supports a pre-build run a script to do some customization before building etc etc.

The problem is that if you try to support Linux/MaxOS with POSIX Shells and Windows you quickly find there is no easy solution that allows your project to build in both without changing the command in the pre-build.

MPLAB simply does the equivalent of system("your command string here") and you are on your own for what happens next.

If you are on a system with POSIX Shells (aka Linux or MacOS) you need to enter something that will run correctly by /bin/sh. I.e. the first thing in the string needs to be either a command (i.e. something found somewhere in $PATH, or a path. The path can be absolute or relative, but it must contain at least one slash ("/"). So either /somewhere/somedir/myscript or ../somewhere/myscript or ./myscript.

If you are on Windows, then again a command found in the $path works. An absolute path works, but only with back slashes ("\"). And if you want to run something out of the local directory (projectname.X for mplab project) you just use the script name. No self relative path name allowed.

So how do you support a project that needs to run without change in both environments? mplabs doesn't help. A quick suggestion would for it to have two script options. One of POSIX systems and one for Windows. But it doesn't...

I found that on Linux (presumably MacOS as well) using sh to run the script worked:

sh -c "./youscripthere your args here"
But how to make that work in Windows?

Solution, a simple program we'll call fakesh. A half dozen lines of .c to look for the argument after -c, strip the "./" and call system() with that:

system("youscripthere your args here")
This works because the first call to system invokes sh.exe (which we put in the project directory.)

The second call to system calls "yourscripthere.bat" which also resides in the project directory.

We end up with the following in the project directory:

  1. sh.exe - the fakesh binary
  2. yourscripthere - posix shell script
  3. yourscripthere.bat - Windows bat file 


For the USBLAN Demo project I use this to generate a confname.h file containing the project name, project dir, device name etc, before creating the new web_pages_img.c file.

And it does work well, for both Linux and Windows, without change. :-)


Fakesh.c


#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

void usage(const char * msg) {
    fprintf(stderr, "Usage: sh -c 'args'\r\n");
    fprintf(stderr, "%s\r\n", msg);
    exit(1);
}

int main(int argc, char *argv[]) {
    char *cp;
    cp = argv[2];
    if (strncmp(cp, "./", 2) == 0) cp += 2;
    system(cp);
}











Thursday, October 31, 2019

Networking over USB versus 100Mbit Ethernet on PIC32MZ EF Starter Kit

This is a quick test using the built in iperf function in the PIC32MZ USBLAN Harmony 3 demonstartion project.

Summary:
  • 83574 Kbps - Networking over USB
  • 94519 Kbps - Ethernet 100Mbit
The USB solution throughput is about 88% of the Ethernet.

The USB data rate utilizes less than 25% of the available bandwidth of the USB High Speed bus (420Mbits).

A closer analysis using Wireshark on the Windows showed that the rate of received frames was almost exactly 7000 and 8000 respectively. Which is the same ratio. I'm assuming that there is an underlying timing issue with respect Microchip Harmony 3 drivers that are limiting the frame rates given that they are both a close multiple of 1000!

During the implementation of the PIC32MZ USBLAN driver, I implemented the ability for frame aggregation which CDC-EEM supports. That allows the driver to check if a transfer is already pending (to be sent) and will if possible then bundle two frames together for the next transfer (this reduces overhead and allows higher throughput in most systems.)

Unfortunately, the PIC32MZ TCPIP stack did not deliver frames fast enough. Or conversely, the USB layer was able to send the data fast enough, so that in all cases single transfers were used. 

I'll need to dig in

Iperf Server (Windows 10)

iperf -s -u -i 10

Iperf Client (PIC32MZ EF SK)

iperf -c 192.168.188.1 -u -b 100M -i 10 -t 60


N.B. the Microchip builtin iperf client only works with the default interface. You may need to set the default interface to the one needed before use:

defnet USBLAN
iperf -c …
defnet PIC32INT
iperf -c ...


Networking over USB


>iperf -c 192.168.188.1 -u -b 100M -i 10 -t 60

iperf: Starting session instance 0

Given in BW: 100000000+0=100000000iperf: Using the default interface!
>    - RemoteNode MAC: f6 0 f5 4c 9c dc
---------------------------------------------------------
iperf: Client connecting to 192.168.188.1, UDP port 5001

iperf: instance 0 started ...
    - Local  192.168.188.2 port 50521 connected with
    - Remote 192.168.188.1 port 5001
    - Target rate = 100000000 bps, period = 0 ms
    - [ 0- 10 sec]   0/ 71153 ( 0%)    83753 Kbps
    - [10- 20 sec]   0/ 71265 ( 0%)    83922 Kbps
    - [20- 30 sec]   0/ 71309 ( 0%)    83973 Kbps
    - [30- 40 sec]   0/ 71302 ( 0%)    83965 Kbps
    - [40- 50 sec]   0/ 71320 ( 0%)    83986 Kbps
    - [50- 60 sec]   0/ 71297 ( 0%)    83959 Kbps
    - [0.0- 60.0 sec]   0/ 427655 ( 0%)    83927 Kbps
    -----------------------------------------
    - [0.0- 60.2 sec]   0/ 427664 ( 0%)    83574 Kbps

Ethernet 100Mbit

>iperf -c 192.168.40.16 -u -b 100M -i 10 -t 60

iperf: Starting session instance 0

Given in BW: 100000000+0=100000000iperf: Using the default interface!
>    - RemoteNode MAC: 70 85 c2 38 a0 d7
---------------------------------------------------------
iperf: Client connecting to 192.168.40.16, UDP port 5001

iperf: instance 0 started ...
    - Local  192.168.40.133 port 55453 connected with
    - Remote 192.168.40.16 port 5001
    - Target rate = 100000000 bps, period = 0 ms
    - [ 0- 10 sec]   0/ 80598 ( 0%)    94876 Kbps
    - [10- 20 sec]   0/ 80619 ( 0%)    94937 Kbps
    - [20- 30 sec]   0/ 80604 ( 0%)    94919 Kbps
    - [30- 40 sec]   0/ 80603 ( 0%)    94918 Kbps
    - [40- 50 sec]   0/ 80610 ( 0%)    94926 Kbps
    - [50- 60 sec]   0/ 80615 ( 0%)    94932 Kbps
    - [0.0- 60.0 sec]   0/ 483660 ( 0%)    94919 Kbps
    -----------------------------------------
    - [0.0- 60.2 sec]   0/ 483669 ( 0%)    94519 Kbps


Thursday, October 24, 2019

Belcarra PIC32MZ USBLAN Harmony 3 demo.

The Harmony 3 server demo project for Belcarra USBLAN is available on pic32mz_usblan_harmony3.


This is a demonstration kit for the Belcarra PIC32MZ USBLAN Driver using either of Microchip's* PIC32MZ EF Starter Kit or PIC32MZ EF Curiosity Board**.

It demonstrates how Networking over USB can be used as an alternative to Ethernet or WiFi to provide lower-cost networking to a Microchip PIC32MZ based board.
Networking over USB allows a point to point network connection to be used between the PIC32MZ board and a Windows, Mac or Linux host.

More information on licensing of the Belcarra USBLAN driver for Windows, MacOS and PIC32MZ is available at https://www.belcarra.com/ or contact info@belcarra.com. 
- **apps/tcpip/web_net_server_nvm_mpfs**
The kit was developed using MPLAB X IDE v5.20.
There are two projects available in the kit:
* pic32mz_ef_sk.X - support for the PIC32MZ_EF_SK Ethernet Starter Kit
* pic32mz_ef_curiosity.X - support of the PIC32MZ_EF_CURIOSITY Board
The sample application was been modified to add the following Harmony 3 components:
* net Telnet
* net Iperf
* net Zeroconf
* USB High Speed Driver
* USB Device Driver
This demonstration can be used with or without the Belcarra USBLAN Driver which is provided as a compiled library.

Supported Microchip Evaluation Boards

- **PIC32MZ EF Ethernet Starter Kit** (pic32mz_ef_sk)
- **PIC32MZ EF Curiosity Board Bundle **(pic32mz_ef_curiosity)

CDC-EEM

Networking over USB uses a USB connection to transfer TCP IP Frames (aka packets) between the USB Device (in this case a Microchip Evaluation Board) and a USB Host (typically a Windows, Mac or Linux system.)
There are various protocols defined by USB.org: * CDC-ECM * CDC-EEM * CDC-NCM
The Belcarra PIC32MZ USBLAN Driver implements CDC-EEM. The CDC-EEM protocol allows for a simple point to point transfer of frames between the USB Host and the USB Device. It supports frame aggregation and his very little overhead.
The USBLAN driver supports zero-copy received frames. I.e. the data is received into a receive buffer by the USB High-Speed driver and that buffer is given to the TCPIP layer for processing.
The USBLAN driver also supports zero-copy transmit frames or can aggregate multiple frames depending on how the data is formatted by the TCPIP lyaer.

Performance

Using iperf the pic32mz_ef_sk board can source TCP data at about 93 Mbits/second with Ethernet and about  83 Mbits/second using USBLAN.

Belcarra PIC32MZ USBLAN Driver

The kit is set up to use the Belcarra PIC32MZ USBLAN Driver. That driver is available under license from Belcarra Technologies. Simply copy the USBLAN driver into src/config/*/driver/usblan.
To use this kit without the Belcarra driver remove the USBLAN definition from the X32-gcc Preprocessing macros and compile.
For information on obtaining a copy of the Belcarra driver contact Email: info@belcarra.com

Microchip Eval Boards

PIC32MZ_EF_SK - Ethernet Starter Kit PIC32MZ2048EFM144 - DM320007-C

Configurations:
- pic32mz_ef_sk_usblan - USBLAN and Ethernet
- pic32mz_ef_sk - Ethernet only
There are two versions of this kit, you may to need to chnage the processor selection to match your kit:
- DM30007 - non-crypto PIC32MZ2048EFH144
- DM30007-C - crypto PIC32MZ2048EFM1444
pic32mz_ef_sk

PIC32MZ_EF_CURIOSITY - Curiosity Board PIC32MZ2048EFM100 - DM320104-BNDL

N.B. the USB UART is shown plugged into Mikro Bus #1. The project configuration requires it to be in Mikro Bus #2.
Configurations:
- pic32mz_ef_sk_curiosity_usblan - USBLAN and Ethernet
- pic32mz_ef_sk_curiosity - Ethernet only

Testing Setup

Use two systems: 1. Linux development and testing for Ethernet 2. Windows for testing USBLAN
Linux setup: - Chrome with two tabs open, for the two possible IP addresses that may be used for the Ethernet (depending on if the Ethernet is the first or second interface). E.g.: http://192.168.40.145/ - Ethernet only configurations http://192.168.40.146/ - USBLAN and Ethernet configurations - Terminal window open running minicom pointing at the appropriate serial port, configured to 460800 baud. Port names may vary, but on my system: /dev/ttyACM0 - PIC32MZ EF SK USB2UART port /dev/tty/USB0 - PIC32MZ EF Curiosity USBUART click board port
Windows setup: - Chrome with one tab open, set to the USBLAN IP address. E.g: http://192.168.188.2/ - Terminal window (e.g. Cygwin mintty). Various tests, but typically: ping 192.168.188.2
The Chrome tabs should display the following screen, with the Random Number being updated rapidly:
Web Server Screen
For the non usblan configurations you will only see the webpage on the Ethernet IP address.
IP address:
- Ethernet is assigned by DHCP from your local DHCP server, see it's DHCP Lease table
- USBLAN will be at 192.168.188.2
On the Starter Kits only buttons 1 and 2 work. Button 3 functionality is lost because the MPU pin it is on is used for the USB2UART Tx line.
On the Curiosity board, there is only one button.

PIC32MZ EF SK - faster programming with MPLAB SNAP

One of the minor annoyances with switching to MPLAB X v5 from v4 is that the programming time for the projects seemed to double. From about 45 seconds to almost 2 minutes.

But v5 MPLAB X supports the new low cost MPLAB SNAP programmer from Microchip. That can program the PIC32MZ EF SK (for my project) in about 17 seconds!

Digikey price is $14.95US: https://www.microchip.com/Developmenttools/ProductDetails/PG164100




To use with the PIC32MZ EF Starter Kit you'll want an adapter that can plug into the 0.05" spacing of the ICSP debug header. See the Olimex adapter for $5 from Digikey: https://www.digikey.ca/product-detail/en/olimex-ltd/PIC-ICSP/1188-1068-ND/3471409





To use with the PIC32MZ EF Curiosity you'll need to add the ICSP debug header. Also, the headers are needed for the Olimex kit. Digikey sells the Microchip TPROG001 cable with headers for $4US: https://www.microchipdirect.com/product/search/all/TPROG001



The SNAP programmer needs to be disconnected after programming to get the PIC32MZ board to reset. To make it easier to do that without stress on the ribbon cables I used two, and an extra header from the TPROG001 kit. With the six wire RJ6 plugs allowing for a simple to connect and disconnect point.







Notes:

  1. The SNAP programmer does not provide power, you'll need to plug the board in (e.g. via USB Debug or USB Device.)
  2. Remove the double header plug at location J2 (near the ICSP header on the PIC32MZ EF SK.)
  3. The six-pin header needs to be plugged into the (looking at the SNAP with the black connector at the bottom) right-hand side of the 8 pin black edge connector. You'll see the small black triangle there.
  4. Set the programming speed to Low (Properties / SNAP / Programming Options).
  5. After programming disconnect the SNAP.





Tuesday, October 15, 2019

Harmony 3 vs Harmony 2 - initial observations


The Harmony 3 Framework is Microchip's current supported environment for development.

Which is not saying much :-(

The sample projects are not nearly as well developed as the Harmony 2 Framework. 
  • There are no sample projects for many of the older boards, e.g. PIC32MX or PIC32MZ_EF_Curiosity
  • Sample projects for Atmel boards are available (Atmel was bought by Microchip in 2016)
  • all networking projects have UART2 support for debugging (good)
  • no networking projects have USB (bad)
  • no USB projects have UART2  (bad)
  • using the Microchip Harmony Configurator 3 to add components to projects effectively broke them, either by making them no compile or not work once compiled (very bad)
Since I need to have a networking project for PIC32MZ_EF_SK that has UART2 and USB working at the same time, that was the first problem.

After about a dozen attempts, I figured out what was being broken to prevent compiles, what was being broken to prevent the minimal projects from working after that. And then managed to get a simple USB function added via MHC3 that didn't break the otherwise working network project. And then managed to get that to enumerate as a USB device without breaking networking....

In theory, I should now be able to add USBLAN to it. And then look at adding support for the PIC32MZ_EC_SK and PIC32MZ_EF_Curiosity boards. 

In some respects the changes to the project setup for Harmony 3 are good. In Harmony 2, projects pointed at the base code in the Harmony 2 library. Which made the configuration xml file a nightmare. As it consistently put in relative file links instead of absolute. Which made publishing for other people problematical, as they would have to have a similar filesystem set up for the projects. Fixing the xml manually was problematic as mplab x would consistently break them again.

In Harmony 3, MHC3 copies the libraries required into the project itself. Which allows each project to be self-contained (except when running MHC3 itself as it needs to be able to find the Harmony 3 files etc.)

In practice, the current MHC3 is buggy in the extreme. Simply opening a sample networking project (e.g. web_net_server_nvms), verifying that it compiles and works and then re-generating the code produces code that does not compile. Some components having been removed (system command and debug).

Fixing those to be able to compile produces something that loads but does not work as the system clocks have been changed in plib_clck.c so that the ethernet module no longer initializes.

Fixing that restored the project to working status.

Then using MHC3 to add USB (e.g. HID Basic), did not add anything to the application layer. I had to dig into the USB device sample projects to copy and integrate the application USB code.

But at least I now have a working PIC32MZ_EF_SK networking project (i.e. httpnet web page functional across Ethernet), with debug output on UART2, and enumerating as a Basic HID device.



pic32mzserverdemo - initial release

The initial release of the pic32mz serverdemo is available via git from bitbucket.

This project supports the three development boards:

  • PIC32MZ_EF_SK
  • PIC32MZ_EF_Curiosity
  • PIC32MZ_EC_SK
  • shows how the Belcarra PIC32MZ USBLAN driver is integrated.


It shows how the Belcarra PIC32MZ USBLAN driver is integrated into a Harmony 2 project that has tcpip enabled.

Debugging output is configured to use the USB2UART on the PIC32MZ_EF_SK and the USBClick board on the PIC32MZ_EF_Curiosity.

For more information and access to the Belcarra driver contact info@belcarra.com.

The server demo project can be compiled and tested with or without the Belcarra Driver.

Using iperf the PIC32MZ_EF_SK board can source TCP data at about 70-80 Mbits/second with Ethernet and about 60-70 Mbits/second using USBLAN.

Sunday, September 29, 2019

IoT and USB Networking

I'll be using this blog to comment on updating Belcarra's USBLAN on the PIC32 and eventually some additional vendors.

We last looked at this in 2015 with the older (pre-Harmony) libraries. It was reasonably easy to get a networking over USB driver implemented. But due to the design of the network stack, difficult to have it available in the same project as Ethernet.

 Microchip introduced a new set of libraries at that time (still pre-Harmony) that looked better. But unfortunately broke the APPIO debug support for the PIC32MX EF SK2. So I was not able to to get either the PIC32MX EF SK2 or the PIC32MZ EC SK working with usable console support to allow debugging.

 Getting back into this project this summer we found that the PIC32MZ EC SK had been replaced witeh the PIC32MZ EF SK. With the primary change being a USB2UART being added to the board to support a serial console.

 There is also a PIC32MZ EF Curiosity board that supports a USB UART click board plugged into one of the two mikro BUS sockets.

 Microchip has introduced a new set of support libraries called Harmony. It appears that the most stable is version 2. Version 3 has been released (I think 2019) but does not support as many of their evaluation boards.