I have installed glassfish5 in a linux machine.
Below are the steps i used to connect using jconsole.
/usr/java/latest/bin/jconsole -J-Djava.util.logging.config.file=/root/logging-jconsole.properties

Copied this - service:jmx:rmi:///jndi/rmi://localhost:8686/jmxrmi in the remote process 

I get the following error -
March26,2018 3:51:46PM RMIConnector Connect
FINER: [ javax.management.remote.rmi.rmiconnector:jmxServiceURL= service:jmx:rmi:///jndi/rmi://localhost:8686/jmxrmi Failed to retreive RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectionIOException: non JRMP server at remote endpoint]

Same steps works for glassfish3 to connect to jconsole.